Skip to content

feat(klaud): park sessions on queued runs and resume when they end / feat(klaud):会话在运行排队时挂起并在运行结束后恢复 - #4474

Merged
adibarra merged 3 commits into
mainfrom
feat/klaud-park-sessions
Oct 24, 2026
Merged

adibarra merged 3 commits into
mainfrom
feat/klaud-park-sessions

Conversation

@adibarra

@adibarra adibarra commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Klaud sessions no longer idle until the 330-minute step timeout while their benchmark runs queue. Last week 61 of 161 sessions died that way, losing their execution logs and turning 36 of 76 planner runs red. Three candidates (#4329, #4340, #4447) exhausted every session on queue time alone.

Changes

  • park command: a session whose remaining work is queued or running records the live runs it waits on (each needs an attempt report) and stops. Only a session past its soft deadline may park without live runs.
  • Stop hook accepts a park while a parked run is still live; it blocks if those runs already ended.
  • PostToolUse hook (check-deadline) tells the agent to report and park at 300 minutes.
  • Recovery never resumes while owned runs are live, reruns the latest preempted final sweep or a newer preempted targeted run, and judges idleness by the candidate's own job, not the whole planner run.
  • Drafts with live runs are kept indefinitely; the 24-hour give-up counts from the later of the last session end and the last run update.
  • Budget: 12 sessions total, at most 4 that end without parking (unchanged from today's 4).
  • Recovery-only wake runs (Klaud Cold wake) at :15 and :45 each hour; planner waves every 3 hours instead of 6.
  • At most 5 Klaud sessions run at once: recovery counts running sessions before resuming, and select gives new candidates only the remaining slots (--active-sessions).
  • A park counts only if the agent ended cleanly right after it; a session that kept working and then ran out stays red and counts as expired.
  • klaud-plan artifacts are kept 90 days so late resume runs can still read them.
  • Parked sessions are green with session-state: parked; structured output is now {"status", "outcome"?} from session-end-schema.
  • Candidate prompt updated to park instead of watching queued work.

Validation
Klaud unit tests and CI lint pass after merging main. A read-only check against live PRs #4447, #4470 and #4471 behaved as expected. First real proof is the next deployed wave: watch for park comments, green parked jobs and wake-run resumes.

中文

Klaud 会话不再在基准运行排队时空等至 330 分钟步骤超时。上周 161 个会话中有 61 个因此终止,执行日志丢失,76 次规划运行中有 36 次显示失败;#4329、#4340、#4447 三个候选仅因排队就耗尽了全部会话。

变更

  • park 命令:剩余工作均在排队或运行时,会话记录其等待的自有运行(每个都需有尝试报告)后停止。只有超过软时限的会话可在无运行时挂起。
  • Stop 钩子在挂起的运行仍在进行时允许停止;若这些运行已结束则阻止。
  • PostToolUse 钩子(check-deadline)在 300 分钟时提示代理报告进度并挂起。
  • 恢复流程在自有运行仍在进行时不再恢复会话,会重跑最新被抢占的最终 sweep 或更新的被抢占定向运行,并按候选自身作业判断空闲,而非整个规划运行。
  • 仍有运行的草稿 PR 会一直保留;24 小时放弃时限从最后会话结束与最后运行更新中较晚者起算。
  • 预算:共 12 个会话,其中最多 4 个未挂起即结束(与当前的 4 个一致)。
  • 每小时 :15 和 :45 运行仅恢复的唤醒(Klaud Cold wake);规划轮次由每 6 小时改为每 3 小时。
  • 同时运行的 Klaud 会话最多 5 个:恢复流程在恢复前统计运行中的会话,select 只为新候选分配剩余名额(--active-sessions)。
  • 只有代理在挂起后正常结束,挂起才计入;挂起后继续工作并超时的会话仍显示失败并计为过期。
  • klaud-plan 产物保留 90 天,以便后续恢复运行读取。
  • 挂起的会话显示为成功并记录 session-state: parked;结构化输出改为 session-end-schema 定义的 {"status", "outcome"?}。
  • 候选提示改为挂起排队工作而非持续观察。

验证
合并 main 后 Klaud 单元测试与 CI lint 均通过。对线上 PR #4447、#4470、#4471 的只读检查符合预期。真正的验证是部署后的下一轮:关注挂起评论、成功的挂起作业以及唤醒运行的恢复。

AI model disclosure

Prepared with Claude Code using anthropic/claude-opus-5-5 (implementation, tests, PR text). Read-only scout subagents on the same model analyzed last week's Klaud runs that motivated this change.

Sessions that wait on owned benchmark runs now park instead of idling until
the 330-minute step timeout. Recovery reruns the newest preempted run, starts
the next session only after every owned run ended, judges idleness per
candidate job, keeps drafts whose runs are live, and runs every 30 minutes
as a recovery-only wake. A PostToolUse hook warns sessions near their limit.

会话在等待自有基准运行时改为挂起,而不是空等至 330 分钟超时。恢复流程会重跑最新被抢占的运行,
仅在所有自有运行结束后启动下一个会话,按候选作业判断空闲,保留仍有运行的草稿 PR,
并每 30 分钟执行一次仅恢复的唤醒。PostToolUse 钩子会提醒接近时限的会话。
@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

inferencex-e2e/infx diff: +645 -152

Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package.

中文

大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要,改动通常也很小。请让你的 agent 清理 PR,使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings marked 🟡 are optional suggestions and need no follow-up push.

default: ''
schedule:
- cron: '0 */6 * * *'
- cron: '15,45 * * * *'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 After merging, Klaud can run many more concurrent agent sessions than the 5-per-wave cap, because resumes dispatched by wake runs are never counted against new candidates. Each wake pass resumes up to MAX_CANDIDATES_PER_RUN sessions. select subtracts only the resumed count from its own wave's recover job (slots = max(0, max_candidates - resumed), main.py:944). Fix: make new-candidate slots account for every live resumed session, for example by having select subtract the live Klaud sessions found by owned_activity, or by capping wake resumes to the headroom left by live sessions.

Why this was flagged

The new 15,45 * * * * cron (klaud-plan.yml:25) runs the recover job 48 times a day, and each run calls recover --max-resumes "$MAX_CANDIDATES_PER_RUN". That can dispatch up to 5 resume runs per pass (lifecycle.py:1166-1186). Wake runs skip review and select, so these resumes never reduce the next wave's slots. When the 6-hourly wave runs, sessions that were ready to resume have already been picked up by earlier wake passes, so its own resumed output is about 0. select then fills all 5 slots with new candidates (main.py:944) while the earlier wake-resumed sessions are still running. On the base branch, resumes happened only in the 6-hourly wave, and select's --resumed made resumed plus new sessions at most 5 per wave. After this change, the number of concurrent Claude sessions and GPU benchmark runs is limited only by how many candidates are open, and with 12 sessions and indefinite retention more of them stay open longer. No other code counts live sessions against the cap: owned_activity is used only by sweep_abandoned.

Verification: klaud-plan.yml:25 adds cron: '15,45 * * * *', on which select is skipped while recover still resumes up to 5. slots = max(0, max_candidates - resumed) at main.py:944 counts only the resumes made by the same run's recover job. Base comparison (f8eceff): recover ran only in the 6-hourly wave, so resumes and new candidates shared the same 5 slots.

Comment thread inferencex-e2e/infx/klaud/__main__.py Outdated
Comment on lines +769 to +770
if not receipt and pulls and resumable(session, pulls[0]):
parked_runs = parked(session, pulls[0], int(os.environ.get("KLAUD_SESSION") or "1"))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 A session that parks, keeps working, and then times out is still recorded as a green "parked" session. It also skips the MAX_EXPIRED_SESSIONS limit. save_diagnostics only checks that the session's park comment exists. It ignores whether the session actually stopped while its parked runs were live, and ignores action_outcome. within_budget makes the same check (not parked(...)), so the timeout is never counted as expired. Fix: a park counts only if the session ended in that parked state. One way is to clear or supersede the park marker when check_stop blocks, or when work resumes after a park. Also require session-end.json status "parked" and a non-timeout action_outcome.

Why this was flagged

The agent runs park on queued runs, which posts the <!-- klaud-park:...:session-N marker (lifecycle.py park). Then either the runs end before it stops, so check_stop blocks with "Every run this session parked on has ended. Continue..." (main.py:658-662), or the agent keeps working instead of stopping. Either way it carries on and later hits the 330-minute step timeout (klaud-candidate.yml timeout-minutes: 330). save_diagnostics (main.py:769-770) finds the old marker and writes session-state "parked". It returns True at main.py:806-818 whatever action_outcome is, so the job goes green. In recovery, within_budget (lifecycle.py within_budget) counts that session as parked because its marker still lists run IDs. The timeout therefore never counts toward MAX_EXPIRED_SESSIONS = 4, which weakens the PR's claim that at most 4 sessions may end without parking. Repeated timeouts like this could use all 12 MAX_SESSIONS. On the base branch every timed-out session counted toward the 4-session cap and produced an unexpected-error diagnostic.

Verification: In save_diagnostics (main.py ~769), when parked is not None and failure is None, it writes session-state: parked and return True, and it never consults action_outcome. So a timed-out session with a stale park marker gets a green job and is recorded as parked. In within_budget, a stale non-empty run list makes not parked False, so the timeout is not counted toward MAX_EXPIRED_SESSIONS=4.

Comment on lines 168 to 180
def resume_runs(self) -> list[dict]:
"""Recovery's resume runs of this candidate's session, oldest first.

A session resumes within RESUME_GIVE_UP_SECONDS of the last ending and lasts at most a
GitHub job's 6 hours, which bounds when every resume run starts.
"""
"""Recovery's resume runs of this candidate's session, oldest first."""
prefix = resume_title(self.parent["id"], self.candidate.id, 0).removesuffix("0")
until = utc(self.parent["created_at"]) + MAX_SESSIONS * timedelta(
seconds=RESUME_GIVE_UP_SECONDS, hours=6
)
runs = self.listing(
"actions/workflows/klaud-plan.yml/runs?"
+ urlencode(
{
"event": "workflow_dispatch",
"actor": BOT,
"created": f"{self.parent['created_at']}..{until:%Y-%m-%dT%H:%M:%SZ}",
"created": ">=" + self.parent["created_at"],
"per_page": 100,
}
),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Long-running candidates can now be resumed after their planner's klaud-plan artifact (kept 14 days) has expired, so every resume run fails and the draft is eventually closed. resume_runs() dropped the old bound on when resume runs start, and MAX_SESSIONS went from 4 to 12. Fix: keep the resume window inside the artifact's life on every path. Either set klaud-plan retention-days to at least the longest possible resume span (90, as klaud-ownership uses), or have resumes() stop and finish once the parent is older than that retention.

Why this was flagged

The old bound in resume_runs() was MAX_SESSIONS * (RESUME_GIVE_UP_SECONDS + 6h) = 4 * 30h, about 5 days. The diff deletes that bound (lifecycle.py:168-181) and raises MAX_SESSIONS to 12. Even with no queue time, 12 sessions at up to 30h each is about 15 days. Every resume run still downloads the parent's klaud-plan artifact (klaud-candidate.yml:76-82), which klaud-plan.yml:308 keeps for only retention-days: 14. Once it expires, the download fails before the gate step, the agent step is skipped and started_session() does not count the job. resumes() then returns True again on every 15,45 * * * * wake pass. grant() reuses the existing grant comment and dispatches another failing run every 30 minutes until the 24h give-up. reconcile() then finishes the candidate as unexpected-error and closes its PR. On the base branch the 5-day bound kept every resume inside the artifact's life.

Verification: The resume job downloads name: klaud-plan with run-id: ${{ inputs.parent-run }} (klaud-candidate.yml:76-83). The planner uploads that artifact with retention-days: 14 (klaud-plan.yml:300-308). The diff drops the base's roughly 5-day bound: "created": ">=" + self.parent["created_at"] (lifecycle.py:177). It raises MAX_SESSIONS = 12.

Comment thread inferencex-e2e/infx/klaud/lifecycle.py Outdated
Comment on lines +1010 to +1028
newest = max(
(
run
for run in runs
if run["conclusion"] != "skipped"
and (run["event"] != "pull_request" or run["head_sha"] == pull["head"]["sha"])
),
key=lambda r: r["created_at"],
default=None,
)
if (
not pending
and latest
and newest
and pull
and pull["state"] == "open"
and latest["conclusion"] == "cancelled"
and now - utc(latest["updated_at"]) < timedelta(seconds=RERUN_GIVE_UP_SECONDS)
and newest["conclusion"] == "cancelled"
and now - utc(newest["updated_at"]) < timedelta(seconds=RERUN_GIVE_UP_SECONDS)
):
await_rerun(session, pull, latest, now)
await_rerun(session, pull, newest, now)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) A preempted final sweep is no longer rerun by recovery once any later targeted run exists, so a candidate the base would rerun and validate needs a new session or gets closed. The code takes the newest owned run overall, then reruns it only if that run is cancelled. A cancelled current-head final sweep with a newer targeted run is skipped. Fix: choose the rerun target among cancelled runs only, so the latest current-head final sweep stays eligible however many targeted runs follow it, as the base did with latest.

Why this was flagged

Trigger: the agent pushes a repair to the PR. Because the changelog entry is already in the diff, run-sweep.yml starts a final sweep on synchronize. The agent then dispatches a targeted e2e-tests run, which passes. The cluster then preempts the final sweep, leaving it cancelled with no failed jobs. In reconcile (lifecycle.py:1010-1028), newest is the max over all non-skipped targeted runs and current-head PR runs. Here that is the passed targeted run, so newest["conclusion"] == "cancelled" is false and await_rerun is never called. latest (lifecycle.py:1029) is the cancelled sweep, so validated is False and reconcile goes to resume. That spends one more session, and the agent has to rerun the sweep itself. If within_budget is already False (4 expired sessions), finish("unexpected-error") closes the PR. On the base, reconcile reran the latest final sweep whenever it was cancelled, up to MAX_SWEEP_ATTEMPTS, before any budget check. The new test only covers a cancelled targeted run that is newer than the sweep.

Verification: nit. The trigger: a current-head final sweep is preempted (cancelled) and a targeted e2e-tests run created after it exists. The rerun gate at lifecycle.py:1020-1028 then requires newest["conclusion"] == "cancelled". If a later targeted run succeeded, await_rerun is never called for the cancelled final sweep. So the base reran this final sweep automatically and this change drops that.

Recovery now counts running Klaud sessions and resumes only while running plus
resumed stay under the cap; select gives new candidates only the remaining
slots. A park counts only when the agent ended cleanly right after parking.
Recovery reruns the latest preempted final sweep even after newer targeted
runs. klaud-plan artifacts live 90 days so late resumes can read them.
Planner waves run every three hours.

恢复流程现在统计运行中的 Klaud 会话,仅在运行与恢复的会话总数低于上限时恢复;
选择阶段只为新候选分配剩余名额。只有代理在挂起后正常结束,挂起才计入。
即使之后有更新的定向运行,恢复流程也会重跑最新被抢占的最终 sweep。
klaud-plan 产物保留 90 天,以便后续恢复读取。规划轮次改为每三小时一次。
@adibarra
adibarra merged commit a42b4a4 into main Oct 24, 2026
5 checks passed
@adibarra
adibarra deleted the feat/klaud-park-sessions branch October 24, 2026 23:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant